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Attachment  A 


Grant  Number:  AFOSR  FA9550-06- 1-0206 

Project  Title:  Modeling  and  Evaluating  Emotion’s  Impact  on  Cognition 

Start  Date  of  Project:  March  1,  2006 

End  Date  of  Project:  April  31,  2009 

ICT  Technical  Point  of  Contact:  Jonathan  Gratch 


Final  Technical  Report 


Proposal  Title(s):  Modeling  and  Evaluating  Emotion’s  Impact  on  Cognition 

Technical  Point  of  Contact:  Jonathan  Gratch,  310-440-0306,  gratch@ict.usc.cdu 

Date:  August  1 1,  2000 

Project  Goals:  The  objective  of  this  project  was  to  make  advances  in  modeling  the  relationship  between 
emotion  and  cognition.  In  terms  of  basic  science,  we  proposed  to  use  eomputational  models  to  conerctize 
psychologieal  theories  eonceming  the  relationship  between  emotion,  cognition  and  behavior,  and  to 
collect  a  body  of  human  performance  data  with  which  to  validate  and  inform  this  process.  In  terms  of 
engineering,  we  proposed  to  construct  a  problem  solving  test  bed  to  which  will  allow  direct  comparisons 
between  human  and  simulated  problem  solving  performance. 

Primary  Accomplishments:  Project  plans  are  largely  proceeding  as  planned.  Aeeomplishments 
include: 

•  Created  a  problem  solving  test  bed  based  to  elicit  and  measure  human  emotional 
responses  and  behavior  tendencies  within  the  context  of  a  two-player  board  game 
(based  on  the  board  game  Battleship,  by  Milton  Bradley). 

•  Modeled  this  situation  within  EMA,  a  eomputational  model  of  human  emotional 
decision  making.  This  model  forms  predictions  of  likely  emotional  reactions  and 
coping  strategies  based  on  an  individual’s  beliefs,  desires  and  intentions  about  a 
situation,  as  well  as  how  these  reactions  and  strategies  change  over  time.  We  then 
used  the  model  to  generate  several  specific  predictions  about  the  nature  and  intensity 
of  human  emotional  responses  to  in-game  events  as  well  as  speeifie  predictions  on 
how  subjects  would  likely  eope  with  these  responses  through  alterations  in  their 
beliefs,  desires  and  intentions. 

•  Developed  an  experimental  methodology  to  test  these  predictions  by  assessing 
subjects’  emotional  responses  in  response  to  carefully  scripted  behaviors  of  a 
confederate  during  game  play. 

•  Developed  a  general  experimental  test  bed  to  study  how  different  communication 
protocols  influences  decision  making  in  multi-player  tasks.  These  protocols  include 
faee-to-faee  communication,  communication  via  video,  via  generated  avatars,  via 
audio,  or  via  text.  Test  bed  is  designed  to  capture  player  game  play  and  a  variety  of 
biometrics  including  vocal  features,  head  movements  and  (to  a  limited  extent)  facial 
expressions. 

•  Carried  out  a  scries  of  experimental  investigations  to  assess  the  validity  of  predictions 
made  by  EMA.  Predictions  were  largely  supported  by  data,  with  some  exceptions 
that  will  motivate  future  study. 

Major  Empirical  Findin2s:  (details  can  be  found  in  the  first  2  publications  listed  below) 

•  A  series  of  studies  investigate  the  accuracy  of  competing  eomputational  models  of 
emotion  in  predicting  the  intensity  of  human  emotional  responses  in  naturalistic 
emotion-eliciting  situations.  The  results  find  clear  differences  in  models’  ability  to 
forecast  human  emotional  responses,  and  provide  guidance  on  how  to  develop  more 
accurate  models  of  human  emotion.  Speeifieally,  expected  utility  provides  an 
excellent  fit  for  modeling  the  intensity  of  the  prospect-based  emotions  (i.c.  hope  and 
fear).  The  fit  for  both  outeome-based  emotions  (i.e.  joy  and  sadness)  is  best 


approximated  by  a  hybrid  model  that  falls  somewhere  between  an  expeeted-utility 
and  threshold  model.  Other  intensity  models  proposed  in  the  literature  did  not  fare 
well.  The  expeetation-ehange  model  was  worse  than  a  degenerate  model  that  ignores 
probability  and  utility.  The  additive  model  performed  better  than  the  expeetation- 
ehange  model  but  was  dominated  by  expected  utility.  The  results  have  clear 
implications  for  computational  models  of  how  human’s  construe  emotionally-eharged 
situations. 

•  A  series  of  studies  investigated  how  people’s  beliefs,  desires  and  intentions  change  in 
response  to  positively  and  negatively  valenccd  emotional  situations.  We  examined 
three  key  kinds  of  coping.  Wishful  Thinking,  Resignation  and  Distancing  that  impact 
an  individual’s  beliefs,  intentions  and  desires,  and  compare  these  coping  responses  to 
related  work  in  the  attitude  change  literature.  We  assessed  hypotheses  derived  from 
the  EM  A  computational  model  of  emotion  against  the  behavior  of  human  subjects 
playing  a  competitive  board  game,  using  monetary  gains  and  losses  to  induce  emotion 
and  coping.  Subject’s  appraisals,  emotional  state  and  coping  responses  were  indexed 
at  key  points  throughout  a  game,  revealing  a  pattern  of  subject’s  altering  their  beliefs, 
desires  and  intentions  as  the  game  unfolds.  The  results  clearly  support  several  of  the 
hypotheses  on  coping  responses  but  also  identified  (a)  extensions  to  how  EMA 
models  Wishful  Thinking  as  well  as  (b)  individual  differences  in  subject’s  coping 
responses.  The  results  have  implications  for  computational  models  of  emotion’s 
influence  on  human  belief-change  and  decision-making. 

Staff  supported  under  grant:  The  following  personal  were  supported  under  the  period  of 
performance  of  the  grant 

•  Faculty; 

o  Jonathan  Gratch 
o  Stacy  Marsella 

•  Postdocs 

o  Wendy  Treynor 
o  Ning  Wang 

•  Graduate  Research  Assistants 

o  Anya  Okhmatovskaia 

•  Undergraduate  Research  Assistants: 

o  JeffMagers 
o  Erin  Margolis 
o  Emily  Salans 

•  Research  Staff 

o  Jamison  Moore 
o  Brooke  Stankovie 

Grant  related  publications  and  invited  talks: 

•  Jonathan  Gratch,  Stacy  Marsella,  Ning  Wang.  Assessing  the  validity  of  appraisal- 
based  models  of  emotion  in  agent-based  systems.  International  Conference  on 
Affective  Computing  and  Intelligent  Interaction.  Amsterdam,  2009 

•  Stacy  Marsella,  Jonathan  Gratch,  Ning  Wang.  Assessing  the  validity  of  an 
computational  model  of  emotion-focused  coping.  International  Conference  on 
Affective  Computing  and  Intelligent  Interaction,  Amsterdam,  2009 


•  Jonathan  Grateh,  Stacy  Marsella  and  Paolo  Petta,  “Modeling  the  Antecedents  and 
Consequences  of  Emotion,”  Journal  of  Cognitive  Systems  Research.  1 0(  1 ),  2009,  pp. 
1-5. 

•  Stacy  Marsella  and  Jonathan  Grateh,  “EMA:  A  Model  of  Emotional  Dynamics,” 
Journal  of  Cognitive  Systems  Research,  10(1),  2009,  pp.  70-90. 

•  Paul  Thagard,  Peter  Ditto,  Jonathan  Grateh,  Stacy  Marsella  and  Drew  Westen. 
Emotional  Cognition  in  the  Real  World.  Proceedings  of  the  Twenty-Ninth  Annual 
Meeting  of  the  Cognitive  Science  Society.  Nashville,  TN;  Lawrence  Erlbaum 
Associates.  2007 

•  Wenji  Mao  and  Jonathan  Grateh.  Modeling  social  inference  in  virtual  agents.  Journal 
of  Artificial  Intelligence  and  Society,  special  issue  on  social  intelligence  and  design. 
24(1),' 2009,  pp.  5-1 1 

•  Paolo  Petta  and  Jonathan  Grateh.  Computational  Models  of  Emotion.  Oxford 
Companion  on  Affective  Sciences.  Sander  and  Scherer  (ed.).  Oxford  University  Press. 

•  Mci  Si,  Stacy  C.  Marsella,  David  Pynadath.  Modeling  Appraisal  in  Theory  of  Mind 
Reasoning.  IVA  2008:  334-347 

•  Invited  panel  organizer  and  speaker.  International  Conference  on  Human-robot 
Interaction,  March  2009 

•  Invited  panel  organizer  and  speaker.  National  Communications  Association,  San 
Diego,  November  2008 

•  Invited  speaker;  Carnegie  Mellon  University,  Human  Computer  Interaction  Institute, 
Seminar  Speaker,  October,  2008 

•  Keynote  speaker.  International  Symposium  on  Ubiquitous  Virtual  Reality,  July  2008 

•  Invited  speaker:  Greifswald  University,  Department  of  Psychology,  July  2008 

•  Invited  presentation;  National  Academy  of  Sciences,  Committee  on  Opportunities  in 
Neuroscience  for  Future  Army  Applications,  February  2008. 

•  Invited  speaker:  USC  Conference  on  Emotion,  Decision  Making,  and  the  Brain, 
August  2007 

•  Invited  presentation;  Simulating  Emotion  and  Nonverbal  Behavior  with  Actors  and 
Agents.  National  Academy  of  Sciences,  Board  on  Army  Science  and  Technology, 
Washington  D.C.,  December  2006. 

•  Distinguished  lecture  series:  Emotion  in  Human-Agent  Interactions.  Technology  and 
Social  Behavior  Speaker  Series,  Northwestern  University,  November,  2006 


abstract 

The  objective  of  this  project  was  to  make  advances  in  modeling  the  relationship  between  emotion  and 
cognition.  The  researchers  proposed  to  use  computational  models  to  concretize  psychological  theories 
concerning  the  relationship  between  emotion,  cognition  and  behavior,  and  to  collect  a  body  of  human 
performance  data  with  which  to  validate  and  inform  this  process.  Major  accomplishments  included  the 
creation  of  an  experimental  testbed  to  study  emotion  influenced  decision  making,  and  a  scries  of 
experimental  studies  using  this  testbed  to  test  specific  hypotheses  produces  by  a  computational  model  of 
emotion-influenced  decision-making.  Results  of  these  studies  lend  support  to  the  validity  of  this 


computational  model.  A  series  of  studies  investigate  the  accuracy  of  competing  computational  models  of 
emotion  in  predicting  the  intensity  of  human  emotional  responses  in  naturalistic  emotion-eliciting 
situations.  The  results  find  clear  differenees  in  models’  ability  to  forecast  human  emotional  responses,  and 
provide  guidance  on  how  to  develop  more  accurate  models  of  human  emotion.  A  series  of  studies  also 
investigated  how  people’s  beliefs,  desires  and  intentions  change  in  response  to  positively  and  negatively 
valeneed  emotional  situations.  We  examined  three  key  kinds  of  coping,  Wishful  Thinking,  Resignation 
and  Distancing  that  impact  an  individual’s  beliefs,  intentions  and  desires,  and  compare  these  coping 
responses  to  related  work  in  the  attitude  change  literature.  The  results  clearly  support  several  of  the 
hypotheses  on  coping  responses  but  also  identified  (a)  extensions  to  how  to  model  Wishful  Thinking  as 
well  as  (b)  individual  differenees  in  subject’s  coping  responses. 


